In recent years, with remarkable improvement of calculation performance and processing speed, and with enhancement of graphic functions for PCs (personal computers), any of display, function and operation in GUI (graphical user interface) become complicated more than ever. Thus, there are many cases where efficient operation is hindered by the complexity as a fetter when an operator operates the computer to perform objective operation.
To improve such situations, presenting and operating information using a three-dimensional space are contrived. This is often called as three-dimensional GUI, which is a mechanism in which an object is three-dimensionally placed in the three-dimensional space, and the object is operated using a predetermined input device. Although, other than this three-dimensional GUI, a similar mechanism may be used in CAD and CG for performing design and the like in the three-dimensional space, explanations are given taking the three-dimensional GUI as an example hereinafter from the point of view of operating and pointing similar three-dimensional objects. By using the three-dimensional GUI, it becomes possible to three-dimensionally place objects that were arranged two-dimensionally or overlapped each other so far. Thus, a working space can be used efficiently. In addition, since the real world surrounding us is the three-dimensional space, by adopting the three-dimensional function into the GUI, the GUI becomes one that can be handled with more perceptiveness compared with the two-dimensional GUI.
Pointing at objects that exist in various depth positions is one problem for operating the three-dimensional GUI. In technologies so far, pointing in the three-dimensional space is realized by adding a new function necessary for depth movement of the pointer, using a mouse, a keyboard, a joystick and the like that have been used for the two-dimensional GUI. However, when using these input devices, since the space in which the operator performs operation with the input device is different from the space where the pointer is actually displayed, there are disadvantages in that it is necessary for the operator to perform additional operation for associating the spaces with each other intentionally, and that the operation itself becomes complicated due to addition of the function for depth movement.
Another problem for operating the three-dimensional GUI exists in operation of objects in the three-dimensional space. In technologies so far, there may be a case in which object operation in the three-dimensional space is performed using a three-dimensional input device such as PHANToM (SensAble Technologies, Inc.) and SPIDER (refer to non-patent document 1, for example). However, like the above-mentioned problem for pointing, since the space in which the operator performs operation using the input device is different from the space in which the pointer is actually displayed, it is necessary for the operator to perform additional operation for intentionally associating the spaces with each other. In addition to that, the space in which the operator can perform operation is limited since an arm and a fixing wire exist. Thus, it is inevitable to use a large-scale apparatus for widening the space. Further, the device is not generally widespread and it cannot be said that the device is familiar to people.
On the other hand, there is a pen-shaped input device as a device having a shape familiar to us. In recent years, there is a pen tablet as one used for pointing and for object operation in two-dimensional GUI. Particularly, an electromagnetic induction type pen tablet (refer to patent document 1, for example) is being adopted in PCs, PDAs (Personal Digital Assistant), and in mobile phones in recent years as an input device that replaces the mouse for reasons that the electromagnetic induction type pen tablet has portability because of its simplicity, and a screen can be directly pointed at with the pen, and that information that can be obtained are abundant (such as two-dimensional position, pen pressure, angle of the case of the pen, states of button and wheel and the like of the pen).
However, there are not many examples in which the pen tablet (pen-shaped input device) that is widely used as the two dimensional input device is used as a three-dimensional input device. As an example, there was a trial in which a pen-shaped device that can obtain its slant and its three-dimensional position is held with a hand in the air, and a display apparatus that exists ahead of the pen virtually displays the pen tip (pointer) so as to realize pointing in the three-dimensional space using the virtually displayed pen tip as a pointer (refer to patent document 2, for example). In this example, although the space in which the operator performs operation is separated from the space in which the pointer is actually displayed, this technology is superior to other conventional technologies since the operator can feel the virtually displayed pen tip almost like a part of a pen held by the operator. However, since the operator must hold the pen-shaped input device with the hand in the air, this technology does not applicable for pointing work for a long time from the viewpoint of exhaustion. In addition, since the hand is in the air, it is hard that the position of the hand stays at one point, so that there is a remarkable disadvantage that the operator cannot hold the pen tip at a target position. That is, since the space in which the operator performs operation with the input device is different from the space in which the pointer and the object are actually displayed, the operator needs to perform additional work for intentionally associating the spaces with each other. Thus, it is substantially difficult to perform detailed operation for objects using this system.
From the above-mentioned viewpoints, a three-dimensional pointing method and a three-dimensional object operation method are required for performing pointing or object operation in the same feeling as in the real world using a three-dimensional GUI that is nearer to the real world than the two-dimensional GUI, in which the apparatus configuration is simple, and the operator can perform pointing and object operation in the three-dimensional space efficiently, accurately and without being fatigued, and processing and editing of the pointed object can be easily performed.
Next, background technologies related to displaying the pointer in the three-dimensional space are described. As described before, in the technologies so far, pointer movement in the three-dimensional space is realized by adding a new function necessary for movement of depth direction to operations by the mouse, keyboard, joystick and the like that are used in the two-dimensional GUI. In addition, the position of the pointer in the depth direction is represented by enlarging or reducing the pointer, for example, so as to realize pointing at an object that is in the back of the space (refer to non-patent document 2, for example). That is, when the pointer is in the front side, the pointer is displayed as a large one, and when the pointer is in the back side, the pointer is displayed as a small one.
However, in many of these methods, the pointer that is displayed in the three-dimensional space can freely move in the depth direction. Thus, even though the pointer is displayed on the display apparatus, there are many cases in which the operator cannot perceive (recognize) the depth position of the pointer so that the operator cannot recognize where the pointer points.
To solve this problem, for example, there is a method for easily perceiving the depth position by displaying a reference, near the pointer, indicating xyz axes of which the depth position does not change (refer to patent document 3, for example). However, since the reference appears on the display plane so that the operator's view is blocked, there may be a case in which working efficiency is lowered. In addition, the reference does not appear when pointing at an object of the real world. Therefore, the method for displaying the reference is not an optimal method from the viewpoint of operating the GUI with a feeling the same as that in the real world.
For reasons mentioned above, for operating the three-dimensional GUI with the feeling the same as that in the real world, an efficient pointing method is required for the operator to be able to recognize quickly and accurately the depth position of the pointer and to recognize where the pointer is pointing, in pointing work using the pointer in the three-dimensional GUI.
In pointing work using a pointer in the conventional three-dimensional GUI, there is a problem in that it is difficult for the operator to intuitively and accurately recognize the depth position of the pointer and recognize where the pointer is pointing.
Next, still another background technology relating to pointer display in the three-dimensional space is described.
As a widely used example of GUI of a computer, a desktop metaphor is known in which an image of a desktop is displayed on a display screen of the computer. Currently, many computers adopt user interfaces using this desktop screen as a core. The desktop screen has been commercialized since latter half of 1980s. Since the processing ability of the computer and the display apparatus are enhanced, fine graphic representation is adopted in the two-dimensional desktop screen. At the same time, the function of the two-dimensional desktop screen is advanced, and the appearance of the screen becomes brilliant, and stereoscopic visual effects become diversified. In addition, recently, a trial is proposed for widening the desktop to the three-dimensional space so as to arrange objects such as icons and windows three-dimensionally (refer to non-patent document 3, for example).
In the three-dimensional desktop, since degree of freedom in the depth direction is added to the conventional two-dimensional plane-like space, there is a merit that the icons and the windows can be arranged functionally. However, even though the desktop is three-dimensionalized, pointing using the pointer is still limited to two-dimensional movement. Therefore, it is difficult to fully utilize the degree of freedom in the depth direction.
On the other hand, it is considered to move the pointer three-dimensionally. However, when the pointer moves three-dimensionally in the space where the object is placed in the three-dimensional space, the pointer is hidden behind the object that is in front of the pointer so that there is a problem in that the operator loses sight of the pointer.
In the desktop interface, it is indispensable that the user always recognize the position of the pointer. In the two-dimensional GUI, it is common that the pointer is always displayed on the foreground. But, when a color of an object at the back of the pointer is the same as a color of the pointer, it becomes difficult to differentiate the pointer from the object, so that it becomes difficult to recognize where the pointer is pointing. For avoiding such problem, it is common to use equal to or more than two kinds of colors for the pointer. When the pointer has colors of black bordered with white, if the color of the background is white, a pointing position can be clearly recognized by the black part, and if the color of the background is black, the pointing position can be clearly recognized by the bordering white part. As mentioned above, in the desktop interface, it is necessary to make the operator always recognize where the pointer is pointing, so that the above-mentioned device is applied in the conventional desktop.
Then, also in the three-dimensional desktop, it is indispensable to make the operator always recognize the position of the pointer. However, in the nature of the three-dimensional space, when the pointer is moved to a position that is in the back (rear side) of an object as seen from the operator, the pointer is hidden behind the object so that there is a problem that the operator loses sight of the position of the pointer. Therefore, it is difficult to recognize where the pointer is pointing, as a result, a fatal problem in the desktop interface in that selection of an object cannot be performed occurs. As mentioned above, even though the desktop is three-dimensionalized, it is difficult to three-dimensionalize the movement of the pointer, so that there is a problem that the merit obtained by three-dimensionalizing the desktop cannot be fully utilized in the three-dimensional desktop.
That is, in the three-dimensional space represented on the display apparatus such as the conventional three-dimensional desktop and the like, there is a problem in that, when a pointer is moved in the three-dimensional space, the pointer is hidden behind the object so that the operator cannot recognize the position of the pointer.
In addition, like the problem that the pointer is hidden behind the object so that the pointer cannot be recognized, in the three-dimensional space represented on the display apparatus such as the conventional three-dimensional desktop and the like, when there is another object (back object) exists in the back of an object (front object), the back object cannot be directly recognized. Thus, for recognizing the position of the back object or for operating the back object, it is necessary to move the front object, or reduce the display region of the front object, or changing the display state of the front object to a hidden state. In addition, when performing the operation to reduce the display region of the front object or to change the display state to the hidden state, for recognizing display information of the front object, it is necessary to perform operation for recovering the display region to the original size and for restoring the display state to the original displaying state. Therefore, there is a problem in that the convenience of the operator is bad.    [Patent document 1] Japanese Laid-Open Patent Application No. 5-073208    [Patent document 2] Japanese Laid-Open Patent Application No. 6-75693    [Patent document 3] Japanese Laid-Open Patent Application No. 8-248938    [Non-patent document 1] Y. Hirata and M. Sato, “3-Dimensional Interface Device for Virtual Work Space,” Proc. of the 1992 IEEE/RSJ Int. Conf. on IROS, 2, pp. 889-896, 1992.    [Non-patent document 2] K. Watanabe, M. Yasumura, “RUI: Realizable User Interface—information realization using cursor”, Human interface symposium 2003, 2003, p. 541-544    [Non-patent document 3] George Robertson, et al. “The Task Gallery: A 3D Window Manage”, Proceedings of CHI2000, 1-6 APRIL 2000, pp. 494-501